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Government policies designed to redress social inequality are often evaluated by qualitative methods, which prevents the establishment 
of a causal relationship between policy objectives and outcomes. Social programmes to broaden participation in higher education have 
been a feature of government policy in Europe, the US and Australia during the past three decades. The principal instrument of policy 
optimisation has been university partnerships with disadvantaged school students. The Australian Labor Government’s Higher Education 
Participation and Partnerships Program (HEPPP) is the most recent iteration of policy to broaden access to university. Unlike many of its 
European counterparts, the Australian policy includes funding criteria for evidence-based partnerships and rigorous evaluation methods. 
In the new era of accountability for publicly funded programmes between universities and disadvantaged schools, the qualitative research 
methods and formative analysis that dominate evaluation research in this held will need to be augmented by quantitative research that 
enables summative evaluation against public policy objectives. Transforming the evaluation paradigm is required to determine the impact 
of university-school partnerships on higher education participation. 


Introduction 

Public policy designed to increase the proportion of 
students from low income backgrounds participating 
in higher education has been introduced in Europe, 
Australia and the US (Woodrow et al. 1998; DEEWR 
2010; Cunningham et al. 2003). Its primary instrument 
of policy optimisation has been university partner¬ 
ships with schools (HEFCE 2009, p. 9; DEEWR 2010; 
Cunningham et al. 2003). The literature indicates 
that the principal objective of these partnership pro¬ 
grammes is to increase the articulation of targeted 
cohorts to university. However, programme evalua¬ 
tions are often designed using qualitative rather than 


quantitative methods, the latter of which is required 
to establish a causal relationship between the pro¬ 
grammes and students’ articulation rate (Cunningham 
et al. 2003; Gale et al. 2010; Passey et al. 2009). The 
qualitative approach has led to the development of 
partnership evaluation instruments that may inform 
incremental improvements in programme delivery, but 
do not enable impact assessment against the principal 
public policy objective. 

In this paper, I propose that a quantitative approach 
to evaluating university partnerships with disadvan¬ 
taged schools and student cohorts is necessary to 
determine whether these programmes improve the 
articulation of low income and low SES students to 
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higher education. The article addresses this proposal 
in four steps: an exploration of the development of 
social programme evaluation as a discipline; observa¬ 
tion of the political challenges associated with evalu¬ 
ation of social programmes; a comparative analysis 
of evaluation methods for university-school partner¬ 
ships in the UK, Australia and the US; and a proposed 
model for evaluating university partnerships with 
disadvantaged schools in alignment with the Austral¬ 
ian Government’s Higher Education Participation and 
Partnerships Program. 

In their 2009 budget paper Transforming Australia’s 
Higher Education System’, the Labor Government set 
a national target that by 2020, 20 per cent of students 
participating in higher education should be from low 
SES backgrounds. The 2010 Higher Education Partici¬ 
pation and Partnerships Program (which regulates the 
20 per cent target), lists principles and objectives for 
university-school partnerships, including evaluation 
criteria set against clear policy objectives. An assess¬ 
ment of the HEPPP objectives and how they might be 
used in the design of research evaluation is discussed 
later in the article. 

The politics of evaluation 

The standard evaluation model for university partner¬ 
ships with disadvantaged schools is social programme 
evaluation’, which arose as a substantial field during 
the New Deal era in North America as a result of vast 
government expenditure on public services (Browne 
& Wildavsky 1987, p. 146). Shadish and colleagues 
define social programmes as those which aim to 
improve the welfare of individuals, organisations, and 
society’ (Shadish et al. 1991, p. 19). Patton observes 
that more rigorous methods of evaluating social pro¬ 
grammes were developed in the wake of the Vietnam 
War when, ‘Great Society programmes collided head 
on with ... rising inflation, and the fall from glory of 
Keynesian economics’ (Patton 1997, p. 11). Evalua¬ 
tion was born of the need for governments to decide, 
among a multitude of choices,‘which things are worth 
doing’(Patton 1997, p. 11). 

It was not any form of evaluation, but that derived 
from quantitative methodology which governments 
began to demand to measure progress against their 
expenditure on social programmes. Posavac and Carey 
identify the policy function particular to quantitative 
methodology as the establishment of causation in pro¬ 
grammes with mass subjects (Posavac & Carey 2007, 


p. 31). For this reason, one might assume that it could 
be used in social programmes also given that they seek 
to establish a relationship between public policy and 
its effect on large groups of people. However, in these 
programmes, including government funded partner¬ 
ships between universities and disadvantaged schools, 
quantitative methods that prove or disprove a causal 
relationship are relatively uncommon. 

Despite the difficulty in evaluating social pro¬ 
grammes, there is an increasing demand for policy¬ 
makers to provide statistical analysis to support their 
continuation or cessation. A 2008 article in Policy 
demonstrates the rationale for establishing evaluation 
based on causal research methods. Farrelly explains 
that randomised trials should be used to test the effi¬ 
cacy of social programmes in order to: 

1. Prove a causal relationship between the program¬ 
matic aims of policymakers and their outcomes. 

2. Develop evidence-based policy. 

3. Use the least random and most scientific method 
available for analysis of social programmes. 

4. Reduce differences in the targeted cohort. 

5. Justify the high ideals and enormous budgets’ of 
social experiments (Farrelly 2008, p. 7-10). 

The research methods described above are used to 
prove or disprove a causal relationship between the 
stated aim/hypothesis of a policy or programme and its 
outcomes (Mark et al. 1999, p. 182).While randomised 
trials provide the most scientifically reliable results in 
research, there is ongoing concern that the motive for 
this form of evaluation is primarily ideological and that 
results can be used by conservatives to discontinue 
public programmes. 

Fisher’s study of the rise of expert policymakers 
in Western post-industrial societies illustrates that 
evaluation research was used to benefit conservative 
political purposes during the Nixon administration in 
the USA (Fisher 1990, pp. 161-163). Nixon institution¬ 
alised the practice of measuring social programmes 
by output measures. Fisher contends that rather than 
being a simple exercise in developing policy exper¬ 
tise in social programming, under the conservative 
approach, ‘low-cost social experiments combined 
with rigid evaluation requirements were often used to 
subvert or eliminate expensive social programs ben¬ 
eficial to Democratic constituencies’ (Fisher 1990, p. 
163). While the characteristics of these programmes, 
such as low government expenditure, undermined 
their chances of success, Fisher asserts that it was the 
methodology used in their evaluation that made an 
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unbiased assessment of efficacy improbable. He con¬ 
tends that this causal research favours conservative 
political approaches to social programmes (Fisher 
1990, p. 163). Weiss further explains that the political 
orientation of evaluation researchers can influence 
how social programmes have been - and are - evalu¬ 
ated. She writes that: 

Because evaluation researchers tend to be lib¬ 
eral, reformist, humanitarian, and advocates of the 
underdog, it is exceedingly uncomfortable to have 
evaluation findings used to justify an end of spend¬ 
ing on domestic social programs. On the other 
hand, it is extremely difficult for evaluators to advo¬ 
cate continuation of programs that they have found 
had no apparent results. The political dilemma is 
real and painful (Weiss 1987, pp. 62-63). 

Weiss describes the 
process of evaluation as 
a rational enterprise, but 
social programmes as 
‘political creatures'.It is the 
tension between rational¬ 
ity and politics which she 
views as the most intransi¬ 
gent obstruction to effec¬ 
tive programme evaluation. 

Despite its development as a distinctive field, from 
the mid 20th century, programme evaluation contin¬ 
ues to receive less of a focus in public policy, including 
educational policy, than implementation, or front end' 
processes. Gerston observes that: 

If there is an underside, or stealth-like component 
to the public policy process, it lies with the proc¬ 
ess of policy evaluation. So much political capital 
is directed towards agenda building, formulation, 
and (to a lesser extent) implementation of a public 
policy, that we often overlook the most obvious 
review questions of all [such as]: did the new policy 
attain its stated objectives along the lines of its 
intentions? (Gerston 2004, p. 119). 

There is, of course, political risk involved in adopting 
quantitative evaluation research methodology as it can 
be used to discredit government funded programmes. 
Social programmes, in particular, which involve large 
groups of people, inevitably involve many variables 
beyond the control of government, policymakers and 
programme managers. For this reason alone, research 
methods that establish a linear path between cause 
and effect may be considered too difficult or too 
resource intensive compared with the kind of public 


policy evaluation described by Gerston, which focuses 
on the design and implementation. However, a review 
of government funded partnerships between univer¬ 
sities and disadvantaged schools illustrates the deficit 
created by evaluation approaches that focus on inputs 
in at the expense of impact assessment. 

The evaluation of university partnerships 
with disadvantaged schools 

An international comparative review of the design of 
university partnerships with disadvantaged schools 
indicates that the connection between universities’ 
programmes and the demands of public policy may 
be poorly understood. While the HEPPP offers clear 
policy objectives, universities have not yet indicated 
what the best evaluation 
method to achieve them 
may be. 

In assessing the evalu¬ 
ation methods of univer¬ 
sity-schools partnerships 
across the UK in 2009, 
Universities UK, the peak 
representative body, estab¬ 
lished that the ‘evidence of 
success’ was defined broadly as follows, Partnerships 
need to deliver results and be measurable in impact, 
particularly through feedback (from students, parents, 
teachers, lecturers, trainers), measurement of progres¬ 
sion and retention of participants’ (Universities UK 
2009, p. 18). However, evaluation based on the meas¬ 
ures of progression and retention of participants in the 
partnership activities and participant feedback is not 
closely aligned with the aim of these partnerships, as 
stated in the foreword of the report, which is to widen 
participation in higher education of participants.There 
is no quantitative evaluation measure suggested for 
such efforts. In their recommendations for progressing 
partnerships programmes, the authors suggested that 
universities might consider benefits from developing 
a consistent approach to assessing impact' (Universi¬ 
ties UK 2009, p. 20). 

The lack of impact evaluation is common among 
university partnerships with disadvantaged schools, 
specifically when impact is aligned with a quantita¬ 
tive policy target such as rate of student participa¬ 
tion in higher education. Aimhigher, funded by the 
UK government’s Higher Education Funding Council 
for England (HEFCE), adopted longitudinal research 
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to assess impact of its university-schools partner¬ 
ships programme. In the 2009 report The longer term 
impact of Aimhigher:Tracking individuals’, researchers 
with the National Foundation for Educational Research 
(NFER) assessed the impact of the programme against 
its objectives, defined in this report as ‘increasing the 
number of young people from disadvantaged back¬ 
grounds who had the qualifications and aspirations 
necessary to enter higher education’ (Morris et al. 
2009, p. 1). 

NFER noted that while the surveys commissioned 
to evaluate the educational outcomes of Aimhigher 
participants recorded aspirations and post-secondary 
destinations, the survey instrument was incomplete 
and left many questions unanswered’. In part, this 
was because in later years, the participant response 
rate was much lower than at the point of first con¬ 
tact and there had been no use of centralised data 
beyond survey results. Questions left unanswered by 
the survey instrument included: 

What were the outcomes for non-respondents? 
What proportion of the various Aimhigher cohorts 
actually went on to higher education? Were there 
any differences in the proportion of young people 
from Aimhigher schools that entered university 
compared to young people from comparison 
schools, where there was no such exposure? Were 
there any differences in the type of universities 
the young people went to - what proportion went 
to those universities that traditionally have higher 
entry requirements, for example? (Morris et al. 
2009, pp. 1-2). 

Many of these questions require datasets that are nei¬ 
ther readily available to higher education institutions,nor 
aggregated by government in a way that could be used 
in quantitative studies. In order to answer them, NFER 
accessed individual student records maintained in the 
National Pupil Database, the Individual Learner Record 
and the Higher Education Statistics Agency. While there 
was an analysis undertaken in 2005 that illustrated that 
Aimhigher had little impact on educational attainment, 
it was conducted almost a decade after the programme 
began. As a result, the findings could not be used to 
in the context of formative evaluation, to contribute to 
programme improvement over that period. 

The peril of not incorporating summative evaluation 
into programme design was also made evident by the 
Aimhigher case. The 2009 research demonstrated that 
there was a statistically significant, but only marginal 
increase of articulation to higher education among 
Aimhigher participants compared with those who 


did not undertake the programme in selected schools. 
There was no impact on the proportion of Aimhigher 
participants attending the most selective universities, 
indicating that other factors, such as overall growth 
in higher education places, may have been important 
determinants in their higher education destinations. 
The finding was reported as evidence of programme 
failure in the Australian media, but perhaps it is more 
specifically an evidence of evaluation failure (Moodie, 
11 November 2009). Indeed, in 2008, the National Audit 
Office had concluded that in relation to Aimhigher, 
government should adopt ‘more robust approaches 
to evaluation when setting up activities which aim to 
widen participation’ (National Audit Office 2008, p. 9). 

The problems associated with poor evaluation of 
university outreach, marketing and recruitment activi¬ 
ties with schools was emphasised by HEFCE in 2008. 
HEFCE criticised the weaknesses in data collection and 
analysis’ in relation to activities included in university 
compacts with government (HEFCE 2008, p. 11). Spe¬ 
cifically, they criticised the lack of student cohort track¬ 
ing and poor evaluation of the impact of engagement 
on access to higher education.They suggested that: 

Better data and analysis would encourage better 
management and enable institutions to make more 
assured judgements about the value of schemes 
and their value for money. It would also begin to 
answer more interesting evaluative questions, not 
just ‘how well have compact participants done after 
entering HE compared with others?’ but ‘how well 
have they done compared with others who share 
similar characteristics and attainment but did not 
participate in a scheme?’ (HEFCE 2008, p. 11). 

The form of evaluation suggested by HEFCE is causal 
research with treatment and non-treatment groups, 
which requires quantitative methodology.As discussed 
later in the article, despite the potential for negative 
political repercussions, policymakers are increasingly 
calling for this form of research to be used in the evalu¬ 
ation of social programmes. 

As with the British case, an Australian analysis of uni¬ 
versities’ outreach to disadvantaged schools in 2009 
found that the evaluation methods used by universi¬ 
ties to assess their outreach to disadvantaged students 
were formative, involving little or no summative evalu¬ 
ation research. Gale and colleagues reported that: 

The most frequently reported program outcome 
was a change in aspirations towards higher educa¬ 
tion. Also commonly reported was an increase in 
students’ understanding of university enrolment and 
procedures. Most respondents reported that their 
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programs are evaluated, predominantly on the basis 

of participant feedback (Gale et al. 2010, p. 8). 

The lack of rigorous evaluation of partnerships in 
Australia may be due to the fact that prior to 2009, gov¬ 
ernment policy on higher education equity included 
neither a national target for the proportion of low 
SES students participating in higher education, nor a 
dedicated fund to drive university partnerships with 
disadvantaged schools. However, rigorous evaluation 
methods should be introduced to determine which 
elements of partnerships contribute most to improv¬ 
ing the academic preparedness of low SES students for 
higher education and to enable policymakers to con¬ 
duct meta-analyses of programmes towards the devel¬ 
opment of an Australian best practice model. 

Gerston’s observation that programme evaluation is 
commonly focussed on the front end of policy develop¬ 
ment rather than programme outcomes is applicable 
to government-funded partnerships between universi¬ 
ties and disadvantaged schools. In their assessment of 
early intervention partnerships between universities 
and schools across the US, Cunningham and colleagues 
concluded that while most programmes did include 
some element of evaluation, it tended to be qualitative, 
and lacked quantitative data that would enable sum- 
mative assessment against policy objectives (Cunning¬ 
ham et al. 2003, p. 29).They proposed that: 

Ideally, indicators of outcomes and effectiveness in 
early intervention programs might include such as 
measures as rates of application to/enrolment in col¬ 
lege, the percentage of participating students taking 
the “pipeline steps” toward enrolment in a four year 
college or university and the percentage of participat¬ 
ing students taking core college preparatory courses 
(Cunningham et al. 2003, p. 27). 

Cunningham et al. concluded that the paucity 
of quantitative data on university partnerships pro¬ 
grammes with disadvantaged schools meant not only 
that progranune effectiveness could not be compared 
in a meta-analysis, but that the individual components 
of these programmes could not be assessed against 
objectives (Cunningham et al. 2003, pp. 36-37). 

Unlike the UK and Australia, Florida’s government 
has established an evaluation model for university 
partnerships with disadvantaged schools based on lon¬ 
gitudinal research that includes qualitative and quanti¬ 
tative data. Its College Reach Out Program (CROP) is 
one of the most established schemes designed with a 
summative evaluation component and was recognised 
by Cunningham et al. as a leader in the field (Cunning¬ 


ham et al. 2003, pp. 36-37). CROP was established by 
the Florida Legislature in 1983 with the objective to 
motivate and prepare educationally disadvantaged, 
low-income students in grades 6 through 12 to pursue 
and successfully complete a postsecondary education’ 
(Winn & Armstrong 2005, p. 7). 

It differs from other university outreach pro¬ 
grammes with schools in two important ways. Firstly, 
since its inception it has had a clear objective which 
was established in legislation. Secondly, its evaluation 
method includes randomised trials and longitudinal 
research towards both continuous quality improve¬ 
ment and assessment against the stated policy objec¬ 
tive. The State also collects aggregated data also to 
analyse broader cohort characteristics and other 
inputs such as participation rates (Florida Postsecond¬ 
ary Education Planning Commission 1994, pp. 8-13). 
As a result of this method of programme evaluation, 
several changes have been made to CROP during the 
past three decades and it has a relatively high rate of 
success against policy objectives. 

While the CROP and HEPPP share some common 
policy objectives, they differ in several important ways. 
Effective evaluation of CROP relies upon accurate 
cohort tracking from point of contact to postsecond¬ 
ary education pass rates. Tracking is enabled by pro¬ 
vision of student social security numbers and Florida 
identification numbers (Postsecondary Education 
Planning Commission 1994, p. 15). Such data is not yet 
gathered in Australia. The CROP students are recruited 
and selected based on low income status and cultural 
diversity is also a prominent factor in selection. The 
targeted selection process enables evaluation against 
random samples of non-participants. In 1994, statutory 
authority for annual evaluation of CROP was given to 
the Postsecondary Education Planning Commission, 
while in Australia, each university is to be responsible 
for evaluation of partnerships. Finally, CROP empha¬ 
sises improving the academic achievement of low 
income students in schools and in postsecondary 
education, rather than focusing on the aspiration-rais¬ 
ing which has become a more prominent feature of 
university-school partnerships in the UK and Australia 
(Winn 2006, p. 1; Gale et al. 2010). 

By 2004, the annual evaluation of CROP demon¬ 
strated significantly higher educational performance 
outcomes of CROP students against a random sample 
of public school students, with the exception of grade 
point average of community college students.Variables 
measured include the progression of school students 
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from one year to the next, the percentage of 12th 
graders receiving standard diplomas, the percentage 
enrolled in higher education, and the percentage of 
community college and state university students with 
a grade point average above 2.0. The evaluation com¬ 
prised 8,286 programme participants and a random 
sample of 10,160 public school students stratified by 
race and income (Winn 2006, p. 2). 

While the Floridian evaluation model is more 
advanced methodologically than that of other coun¬ 
tries, there is an opportunity for Australia to develop a 
rigorous form of social programme evaluation through 
the Higher Education Participation and Partnerships 
Program. Under paragraph 1.80.5(f) of the HEPPP 
guidelines, the Government clarifies universities; 

... will be required, as part of their Partnership 
programs, to provide an ‘evidence base’ for pro¬ 
posed programs. This will need to include intended 
program outcomes, methods for achieving these 
outcomes, and associated measures for tracking 
outcomes. For experimental and pilot projects, 
providers will need to demonstrate how the pro¬ 
gram will achieve the objectives outlined at section 
1.70.1 and the principles outlined at section 1.80.5’ 
(DEEWR 2010, p. 18). 

As illustrated, the guidelines require evidence based 
evaluation of programme outcomes against established 
policy objectives. The new criteria for partnerships 
funding evince a significant change in government reg¬ 
ulation of student access and equity activities in uni¬ 
versities. Under the previous legislation - the Equity 
Support Program - there was no demand for evidence- 
based evaluation of university-school partnerships. 

Fit For Purpose: A Mixed Method Approach 
to Evaluation Research 

The proposed approach to assessing the effect of 
university-schools partnerships is a mixed method 
research design that produces quantitative and qualita¬ 
tive data to enable summative and formative evaluation 
of programmes.The reason for selecting this approach 
is that it addresses most readily the logic underlying 
government funding of university partnerships with 
disadvantaged schools, as articulated in the objectives 
and principles of the HEPPP. 

Several scholars have observed that pragmatism’ 
has been a core motivation for the evolution of mixed 
method evaluation research, which combines quan¬ 
titative and qualitative methodologies (Tashakkori & 
Teddlie 1998; McConney, Rudd & Ayres 2002, p. 122). 


Greene traces the history of mixed methods back to 
the 1980s and explains that social scientists in the 
‘highly practical fields’ such as ‘education, nursing, 
and especially evaluation’ have been using a variety 
of evaluation methods because ‘the contexts in which 
they worked called for both generality and particu¬ 
larity’ (Greene 2008, p. 7). However, she argues that 
theoretical and epistemological concerns have also 
driven the evolution of mixed methods research in 
more theoretical fields such as development econom¬ 
ics (Greene 2008, pp. 7-8). In their 2009 book Mixed 
Method Design:Principles and Procedures , Morse and 
Niehaus offer a comprehensive treatment of mixed 
method design.They explain: 

Our definition of mixed methods is that the study 
consists of a qualitative or quantitative core compo¬ 
nent and a supplementary component (which con¬ 
sists of qualitative or quantitative research strategies 
but is not a complete study in itself). The research 
question dictates the theoretical drive, as either 
inductive or deductive, so that the onus is on the 
researcher to be versatile and competently switch 
inductive and deductive positions according to the 
need of the study ... We have presented our view 
- it appears to work, appears to pass validity tests, 
and gets one to where one is trying to go (Morse & 
Niehaus 2009, p. 20). 

This definition offers a partial solution to the main 
argument against mixed methods; the perceived 
incompatibility between quantitative and qualitative 
paradigms and the resultant confusion about how to 
combine them in a single research project (Morse & 
Niehaus 2009, p. 19). However, Morse and Niehaus 
identify a major problem with mixed method research 
design: there is no consensus about how to evaluate 
mixed methods’ (Morse & Niehaus 2009, p. 20). It is 
observable in their definition of mixed methods that 
the methodology may produce a repeated emphasis 
on the front end and implementation process com¬ 
ponents of policy to the subordination of testable 
outcomes against policy objectives. It is exactly this 
problem that, as discussed throughout this article, has 
undermined the political and scientific legitimacy of 
social programmes for the past half century. 

The rise of mixed methods research has produced 
an attendant need to clarify not only its theoretical 
purpose in relation to evaluation, but its formal meth¬ 
odological structure. A useful distinction is the delinea¬ 
tion between different purposes of evaluation made by 
Nevo. He explains that, evaluation can serve two func¬ 
tions, the formative’ and the ‘summative.’ In its forma- 
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tive function evaluation is used for the improvement 
and development of an ongoing activity (or program, 
person, product, etc.) In its summative function, evalu¬ 
ation is used for accountability, certification, or selec¬ 
tion’(Nevo 1983, p. 119). 

Since that early definition, evaluation epistemology 
has become more complex as scholars attempt to find 
a model that allows for different research requirements. 
In their analysis on the integration of qualitative and 
quantitative research, Tashakkori andTeddlie conclude 
that while integration may occur at any time during 
the research project, ‘true mixed method designs have 
clearly articulated mixed research questions, neces¬ 
sitating the integration of qualitative and quantitative 
approaches in all stages of the study’ (Tashakkori & 
Teddlie 2009, p. 284). However, like Morse and Niehaus, 
they allow that elements of a study may have research 
questions that are predominantly qualitative or quantita¬ 
tive (Tashakkori &Teddlie 2009, pp. 284-285). 


To distinguish fully integrated from partially inte¬ 
grated methods, Tashakkori and Teddlie categorise 
different types of mixed designs create a separate sub¬ 
category called quasi-mixed designs (Tashakkori & 
Teddlie 2009, pp. 288-289).Within mixed designs, they 
identify five types of methods: parallel, sequential, con¬ 
version, multilevel and fully integrated (Tashakkori & 
Teddlie 2009, p. 289). Parallel mixed designs are those 
in which there are at least two interconnected strands, 
each requiring either qualitative or quantitative ques¬ 
tions, data collection and analysis. The strands remain 
either purely qualitative or quantitative, that is, parallel 
(Tashakkori & Teddlie 2009, p. 289). Sequential designs 
are those in which there are at least two strands 
occurring chronologically with mixed qualitative and 
quantitative questions and analysis resulting from the 
findings of the previous element (Tashakkori & Teddlie 
2009, pp. 289-290). Conversion designs involve mixing 
qualitative and quantitative approaches at all stages. 


Table A: Mixed Method Evaluation Model for the Higher Education Participation and Partnerships Program 


HEPPP Objective 1.70.1 

Research 

Method 

Activity Type/s 

Evaluation 

Instrument 

Performance 

Indicator 

Outcome 

a. Assist in improving 
the understanding and 
awareness of higher 
education as a viable post¬ 
school option 

Quahtative 

Master classes 
that introduce 
students to and 
famiharise them 
with university 
curriculum. 

Survey 

Aspirations and 
awareness raised 

Proportion of targeted cohort 
with increased awareness 
above an established 
knowledge threshold. Increase 
in proportion of targeted 
cohort aspiring to higher 
education before and after 
intervention. 

b. Assist in pre-tertiary 
achievement, either at 
school or via an alternative 
pathway, to enable 
consideration for access to 
higher education 

Quantitative 

Early intervention 
literacy and 
numeracy 
mentoring 
programme. 

Pre and post 
activity testing 

Test results 

Improved Uteracy and/or 
numeracy based on pre and 
post testing. 

c. Encourage an increase 
in the proportion of 
[people from low SES 
backgrounds] who 
apply for attendance at a 
provider 

Quantitative 

Admissions & 
scholarships 
workshops to 
assist students to 
navigate pathways 
to higher 
education. 

Application 

cohort 

tracking 

AppUcation rate 
of low SES cohort 
targeted compared 
with non-low SES 
cohort in a given 
year. 

Increased appUcation rate 
of targeted cohort against 
random sample. 

d. Support [people from 
low SES backgrounds] 
in Unking with higher 
education providers. 

Quantitative 

On campus 
activities. 

Tracking 
of number 
of ‘Unking’ 
activiUes held 
with targeted 
cohort against 
random 
sample. 

Proportion of 
targeted cohort 
participating in 
activities and 
proportion making 
contact with HE 
provider. 

Proportion of targeted cohort 
participating in activities and 
making contact with higher 
education providers is equal 
to - or above - the rate of the 
random sample. 


*’Activity type lists only one activity for the purpose ofsimplification. There are many examples that could be listed against each objective. 
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The rule guiding this approach is that when one type 
of data (either qualitative or quantitative) is collected, 
it is analysed using the other type of approach (Tashak- 
kori & Teddlie 2009, p. 290). The multilevel and fully 
integrated designs are progressively more integrative. 

A mixed method evaluation model for 
university-school partnerships 

For the purpose of evaluating university partnerships 
with disadvantaged schools, it is proposed that the par¬ 
allel mixed design would be the most useful.The con¬ 
version, multilevel and fully integrated approaches are 
more experimental than logical at this stage and would 
thus not be easily testable against policy objectives. 
The sequential design has been rejected because of 
its requirement for chronological sequencing, which 
would make coherent longitudinal research of any 
large and diverse social programmes, including exist¬ 
ing university-school partnerships, difficult. The par¬ 
allel mixed design produces contains three elements 
that are useful to analysis of university partnerships 
with disadvantaged schools against the requirements 
of the HEPPP and social programmes more broadly: 

1. A combination of causal and descriptive research 
design and thus the possibility of testing results 
against a core hypothesis (the main policy objective 
underpinning the programme) and sub-hypotheses 
(the programme components). 

2. Qualitative and quantitative data generation. 

3. Summative and formative assessments that can 
be used to assess individual programmes against 
institutional performance indicators, improve pro¬ 
gramme design and implementation incrementally; 
and compare programmes in a meta-evaluation for 
national policy development. 

A simplified example of the parallel mixed design 
applied to the HEPPP is presented in Table A. To close 
the gap between government policy objectives and 
social programme implementation, I have selected the 
principles and objectives of the partnership compo¬ 
nent of the HEPPP as the basis for the evaluation pro¬ 
gramme. Under the Partnership component B, Section 
1.70, activities are to be directed towards domestic 
undergraduate students from a low SES background 
(DEEWR 2010,p. 16). 

While Table A provides an example of how a mixed 
method research and evaluation design might be 
employed to measure progress against the partnership 
objectives of the HEPPP, the question remains as to 


E R S I T I E S 

how the results of the mixed model could be used to 
assess whether the HEPPP, or any equivalent social pro¬ 
gramme, were successful at the level of national policy. 
This is because there is yet to be devised a method 
for synthesising data derived from mixed methods 
research that provides for meta-analysis. The only way 
in which a meta-analysis of HEPPP partnerships could 
be undertaken would be to select a specific subset 
of partnership activities that evidence demonstrates 
are likely to have the strongest impact on the princi¬ 
pal policy objective of increasing low SES students' 
articulation to higher education and test them in a 
randomised trial across schools. I suggest that for the 
purpose of evaluating the HEPPP at the policy rather 
than programme level, one of two approaches to meta¬ 
analysis might be considered: 

1. Only the results arising from the quantitative meth¬ 
ods are measured in meta-analysis for the purpose 
of determining efficacy against the principal objec¬ 
tive of the HEPPP, which is to increase the rate of 
low SES students’ participation in higher education. 

2. Quantitative methods are used to develop national 
policy, while the results from mixed methods are 
used to inform institutional level improvement in 
partnerships programme design. 

Conclusion 

The evolution of evaluation as a research field raises 
new challenges for policymakers and managers of gov¬ 
ernment funded programmes. The question of how to 
measure mass programmes against single, or multiple 
policy objectives demands awareness not only of eval¬ 
uation research methodologies, but how the politics 
of evaluation may affect the ability to implement desir¬ 
able models. 

In the field of social programmes, the politics of eval¬ 
uation has been an impediment to the introduction of 
a linear model of evaluation research that establishes a 
causal relationship between the original policy objec¬ 
tive and its programmatic outcomes. Mixed method 
research offers a model to bridge the philosophical 
and instrumental divide between quantitative and 
qualitative methodologies. Perhaps more importantly, 
mixed methods allow for an understanding human 
variability which, rather than necessarily being a 
signal of programme failure, may, in some cases, be an 
opportunity for progression. The proposed model for 
measuring partnerships established under the Austral¬ 
ian government’s Higher Education Participation and 
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Partnerships Program demonstrates a mixed method 
approach to evaluation. This method will inform pro¬ 
gramme efficacy against the principal policy objec¬ 
tive of the HEPPP and enable formative assessment to 
improve programme design over time.The creation of 
a meta-analytical evaluation model for public policy 
such as the HEPPP is critical to furthering evaluation 
research in the field of social programmes. 

Jennifer Oriel is Manager, Student Access and Equity at 
Deakin University, Australia. 
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